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An extensive statistical survey of universal approximators shows that as the dimension of a typical 
dissipative dynamical system is increased, the number of positive Lyapunov exponents increases 
monotonically and the number of parameter windows with periodic behavior decreases. A subset of 
parameter space remains in which topological change induced by small parameter variation is very 
common. It turns out, however, that if the system's dimension is sufficiently high, this inevitable, 
and expected, topological change is never catastrophic, in the sense chaotic behavior is preserved. 
One concludes that deterministic chaos is persistent in high dimensions. 
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Physical theory attempts to describe and predict the 
natural world by expressing observed behavior and the 
governing balance of forces formally in mathematical 
models — models that can only be approximate represen- 
tations. Empirically, many natural phenomena persist 
even when control parameters and external conditions 
vary. For example, the essential character of fully de- 
veloped fluid turbulence is little affected if one slightly 
changes the energy flux that drives it or if a small dent is 
made in the containing vessel's wall. In building a theory 
of a system exhibiting this kind of dynamical persistence, 
one hopes that, despite its approximations, one's model 
also has this persistence. 

A century of analyzing nonlinear dynamical systems, 
however, has lead to an apparent inconsistency with this 
goal. Since the days of Poincare's development of qual- 
itative dynamics, mathematicians and physicists have 
probed differential equations to test their solutions for 
different kinds of stability. Poincare's discovery of de- 
terministic chaos "16^ demonstrated that at the most de- 
tailed level, there was inherent instability of system solu- 
tions: change the initial condition only slightly and one 
finds a different state-space trajectory develops rapidly. 
Later studies showed that there was also an instability 
in behavior if the equations or parameters were changed 
only slightly H [l^. Even arbitrarily small functional 
perturbations to the governing dynamic leads to rad- 
ical changes in behavior — from unpredictable to pre- 
dictable behavior, for example. The overall conclusion 
has been that nonlinear, chaotic systems are exquisitely 
sensitive, amplifying arbitrarily small variations in initial 
and boundary conditions and parameters to macroscopic 
scales. 
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How can one reconcile this with the observed fact 
of dynamical persistence in many large-scale systems? 
We take dynamical persistence to mean that a behav- 
ior type — e.g., equilibrium, oscillation, chaos — does not 
change with functional perturbation or parameter vari- 
ation. Here we describe the results of a Monte Carlo 
survey which empirically demonstrate that chaos is dy- 
namically persistent if the dimension of a nonlinear sys- 
tem is sufficiently high. More constructively, we argue 
that a particular mechanism is responsible for persistent 
chaos. 

Specifically, the survey shows that in large-scale sys- 
tems dynamical sensitivity — when defined as breaking 
topological equivalences associated with structural sta- 
bility fl^, ergodicity [rP|, and statistical stability — is 
typically benign and does not affect behavior types. Nat- 
urally, drastic changes in the invariant measure yield dif- 
ferent observed dynamics, but the results indicate that 
this becomes increasingly less probable. Moreover, the 
instability associated with deterministic chaos dominates 
high-dimensional dynamical systems, except at extreme 
parameter settings. 

Much of the intuition and motivation for our investiga- 
tion comes from the analytical results found in abstract 
dynamical systems theory, but our construction and con- 
clusions highlight a distinct difference. Said most simply, 
the number of dimensions of the dynamical system mat- 
ters. That is, there is a qualitative difference between 
common behaviors in high- and low-dimensional dynam- 
ical systems. Beyond giving empirical evidence to sup- 
port these conclusions, we introduce a definition of per- 
sistent chaos that suggests an alternative approach to the 
long-standing questions of dynamical stability and offer 
a mathematical conjecture on the mechanism underlying 
persistent chaos in high dimensions. 

The spectrum of Lyapunov characteristic exponents 
(LCE) Q will be our primary tool for analyzing and 
identifying behavior types since there is an equivalence 
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between the number of negative and positive Lyapunov 
exponents and the number of global stable and unsta- 
ble manifolds, respectively — structures that organize the 
state space and constrain behavior Therefore, in re- 
ferring to topological variation here we mean a change in 
the number of positive Lyapunov exponents. 

In order to give a complete representation of the space 
of all systems, we investigate typical behaviors in high 
dimensions using a class of dynamical systems that are 
known to be universal function approximators. (They 
are universal in that t hey approximate arbitrarily closely 
any C mapping, cf. [lOj.) These are single-layer neural 
networks of the form 



Xt = I3q + Pi tanh s 




(1) 



which are maps from R'^ to R. Here n is the number of 
hidden units (neurons), d the number of time lags which 
determines the system's input (embedding) dimension, 
and s a scaling factor for the connection weights Wij . The 
initial condition is (xi, X2, ■ ■ ■ , Xd) and the state at time t 
is {xtT^Xt+i, . . . , Xt+d-i)- The approximation theorems of 
Ref. 01 and time-series embedding results of Ref. 0| 
establish an equivalence between these neural networks 
and general dynamical systems; cf. 0|- 

In the Monte Carlo survey we sample the {n{d+l) + l)- 
dimensional parameter space taking (i) Pi e [0, 1] uni- 
formly distributed and rescaled to satisfy Pi — "-i 
(ii) Wij as normally distributed with zero mean and unit 
variance (which is adjusted with s), and (iii) the initial 
Xj G [—1,1] as uniform. These distributions — denoted 
TTip, fTiw{s), and 771/ — form a product measure on the 
space of parameters and initial conditions — the survey's 
results then are statistical estimates with respect to this 
product measure. We use the s parameter as the pri- 
mary control as it gives the magnitude of the argument 
of tanh(a;). When x « Eq. is linear and one finds 
fixed points and limit cycles; when |a:;| S> 1 the output is 
binary and one finds 2" different periodic states; and for 

a; I between these extremes we find the nonlinear behav- 
ior we will focus on. 

We define persistent chaos in terms of the LCE spec- 
trum as follows: 

Definition 1 (Degree-p Persistent Chaos) Assume 
a discrete-time map f that takes a compact set to itself. 
The map has persistent chaos of degree-p if there exists 
an open subset U of parameter space, such that, for all 
S, 'E U and a given open set O of initial conditions, f\^ 
retains p > 1 positive Lyapunov exponents. 
Persistent chaos (p-chaos) of degree p is our notion of dy- 
namical equivalence on an open set of parameter space. 
This differs from that of a robust chaotic attractor of 
Refs. ,4j and 5], for example, in that we do not re- 
quire the attractor to be unique on the subset U. This 
is an important distinction since, physically, there is lit- 
tle evidence indicating that such strict forms of unique- 
ness are present in many complex physical systems .12j 
and, technically, uniqueness is significantly more difficult 




FIG. 1: LCE spectrum as a function of scale factor s for a 
network of 32 neurons and 64 dimensions. (15000 total time- 
steps; 5000 initial time-steps to arrive on the attractor.) 



to demonstrate. Indeed, alternative dynamical equiv- 
alence frameworks — such as nonuniform partial hyper- 
bolicity — were invented to circumvent problems with 

nonuniqueness . 

Figure [1] represents the typical scenario for the LCE 

spectra 21] of the high-dimensional neural networks as a 
function of s. Here typical refers to what was observed 
in > 99% of the 15800 networks with 77 > 32 and d > 32 
given 771^, mw{s), and mj as defined earlier. Important 
features to notice include the lack of periodic windows 
and that the LCEs vary continuously with s and have 
a single maximum (up to numerical fluctuations). The 
survey also reveals that the maximum number of positive 
Lyapunov exponents is approximately d/A and the at- 
tractors' Kaplan- Yorke dimension is roug hly d/2 ^. As 
d increases, the length of the s-intervals between LCE 
zero-crossings decreases as ^ d^^'^^. These properties 
contrast sharply with the familiar low-dimensional sce- 
narios where one typically encounters a preponderance 
of stable behavior and periodic windows and the LCEs 
vary in a discontinuous manner with control parameters. 
(A more complete analysis of these observations is found 
in Ref. [^.) Moreover, this scenario is different from 
low-entropy, spatially-extended systems discussed in 
or 11]. In our systems, the effective dimension is irre- 
ducibly high and the uniformity of the dynamics is more 
robust. 

These observations complement those from a previous 
study of chaos in neural-network continuous-time differ- 
ential equations j20|. There, a mean-field analysis, which 
assumes that inputs are statistically independent (and 
which does not apply in the present case), also suggested 
that chaos should be common in high dimensions. 

In light of Fig. and the fact that LCE zero cross- 
ings become asymptotically dense {\Ui\ ~ d~^-^^ for 
i < 20, c? ^ 128) we propose the following dynamical 
mechanism for persistent chaos. For a finite but arbi- 
trarily large number of dimensions along an s interval — 
e.g., s e (2,8) — there is an asymptotically dense, always 
countable sequence of parameter values that have an LCE 
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transversally crossing through zero. Thus, a continuous 
path along an s-interval yields inevitable, but noncatas- 
trophic (i.e. p > 1) topological change. This implies 
that when varying parameters, periodic and quasiperi- 
odic windows will not exist in chaotic regions of param- 
eter space of dynamical systems with a sufficiently large 
number of positive exponents (i.e., when entropy rate is 
large). The lack of dense periodic and quasiperiodic win- 
dows is a necessary condition for p-chaos. 

To test this picture, we analyzed the existence of pe- 
riodic and quasiperiodic windows along s € (1,4) in net- 
works with n = 32 and d ranging from 8 to 128 and with 
an ensemble of 700 networks per n and d. We observed 
that (i) the mean fraction of networks with periodic and 
quasiperiodic windows decreases like ~ d~^'^, (ii) the 
mean number of windows decreases like '~ d~^, and (iii) 
the window lengths increase linearly with increasing d. 
These observations are insensitive to increments in s as 
long as As < 0.005. As the dimension increased above 64 
the only networks with periodic windows had windows 
that persisted for most of the s-interval under consid- 
eration. That is, as dimension was increased, periodic 
windows became increasingly rare. When they were ob- 
served, however, they were neither small nor intermit- 
tent, but instead dominated the dynamics. 

To explore the full parameter space systematically, one 
can fix s and vary the weights with random perturbations 
of a given size. We surveyed networks with parameters 
varied in a {n{d -|- 1) -I- l)-ball with its center fixed at s 
noting that the results are insensitive to s variation in 
the chaotic portion of the s-interval; i.e., nearby s val- 
ues yield identical results. Similarly, the results are in- 
sensitive to perturbation size — weight variation on scales 
ranging from 10"^'^ to 1 yield similar results. 

The results are shown in Fig. |21 the probability of 
observing periodic windows decreases as the dimension 
increases. Each data point corresponds to the probability 
of finding a system with a periodic orbit among a set of 
700 networks at a given n and d and each perturbed 
100 times. The range of weight perturbations was 10"'^ 
with s = 3. We found that the probability of periodic 
networks decreases as d~^. Thus, as dimension increases 
the systems are far less likely to display periodic windows 
and, as a consequence, become more persistently chaotic. 

While this is strong evidence for the disappearance of 
periodic windows in parameter space with increasing di- 
mension, a stronger argument follows from our observa- 
tion that the fraction of networks with windows decreases 
less quickly (~ d^^'^) than the overall probability of win- 
dows (~ d~^). Thus, periodic windows which do exist are 
concentrated in an ever-decreasing fraction of networks 
and those with one periodic window are more likely to 
have many periodic windows. 

To quantify the degree of p-chaos for an ensemble of 
mappings with our construction consider the distribution 
of LCE's at a fixed s, noting that p depends on s. If one 
takes p to be the mean number of positive LCE's, then 
p increases monotonically like d/4 P| when s is set 
to give the maximum number of positive LCEs. If one 
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1 2 3 4 5 6 7 8 

FIG. 2: Log probability of periotec behavior versus log di- 
mension for 700 cases per d. Each case has all the weights 
perturbed on the order of 10^'^; 100 times per case. The 
best-fit line is ~ l/d^. 



considers p to be the mean number of positive LCE's mi- 
nus 3 standard deviations, a more conservative estimate, 
p > at d = 32 and increases like '-^ 4 log d at s = 3 Q . 
We will refrain from arguing for a best definition of p and 
simply note that p increases with d monotonically. To- 
gether with the periodic window data, this implies that 
while the parameter change required to alter the absolute 
number of positive exponents decreases, the perturbation 
required for all the positive exponents to vanish grows 
substantially. Thus, the chance of falling into a periodic 
window vanishes in proportion, and chaos becomes per- 
sistent over a considerable portion of parameter space. 

These observations and detailed analysis of 400 four- 
dimensional dynamical systems and 200 64-dimensional 
dynamical systems, as well as many of intermediate di- 
mension, leads to the following view of dynamic (topolog- 
ical) variation with parameter change. All of the LCEs 
that become positive are negative for very small and very 
large values of s — the LCE spectra exhibit a single max- 
imum. As the dimension d is increased, their variations 
decrease and their s-dependence becomes smoother; re- 
call Fig. n Moreover, with increasing dimension the 
number of positive exponents increases monotonically 
m. Finally, the distance between LCE zero-crossings, 
above the maximum, decreases with dimension as shown 
schematically in Fig. 

Figure |3| is a graphical depiction of the hypothesized 
persistence mechanism — a plot of the s axis transversally 
intersected by LCEs. In sufficiently high dimensions, the 
subsets Ui shrink and eventually fall below any resolvabil- 
ity. The result, then, is twofold: one observes continuous 
topological change (bifurcations) , but this is never catas- 
trophic. One sees persistent chaos of varying degrees. 
The onset of sufficiently high dimension for this to occur 
for our dynamical systems was observed to be d > 30. 
These investigations lead to the following conjecture for 
persistent chaos in high dimensions: 

Conjecture 1 Assume f as in Eq. 0) and a sufficiently 
large number d of dimensions and number k = n{d+2)-\-l 
of parameters. There exists a large pi / Lebesgue measur- 
able set of s € with respect to mp, mw{s), and mj, 
for which chaos will be degree-p persistent. Moreover, 
p — > oo as d ^ oo. 

Since our networks are universal function approximators. 
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FIG. 3: Lyapunov spectrum versus network nonlinearity s; 
I7i's are the open sets in parameter space where structural 
stability is believed to persist. The |?7i| parameter intervals 
shrink like ~ as the dimension increases. 

this behavior should be observed in typical nonlinear 
high-dimensional dynamical systems. 

Two comments are in order. First, the existence of 
chaos as a persistent behavior type depends on dimen- 
sion. The subset of parameter space in which chaos 
becomes persistent increases in size (with respect to 
Lebesgue measure) as the dimension of the dynamical 
system increases. This is due both to the increase in the 
number of positive LCEs (given a sufficient increase in 
n) and to a decrease in the appearance of periodic win- 
dows. Second, persistence is related to the number of 
(linearly independent) parameters in the dynamical sys- 
tem. The number of neurons in the network effectively 
controls the entropy rate |3| — that is, increasing the 
number of neurons increases the entropy rate, number 
of positive exponents, and the maximum of the largest 
exponent. Increasing n simply increases the degree (p) 



of the persistent chaos, but the mechanism for persistent 
chaos remains, due to the decreasing probability of pe- 
riodic windows. Networks with few parameters exhibit 
considerably less persistent chaos. 

In this way high entropy-rate systems are more persis- 
tent with respect to functional and parameter perturba- 
tions. This is in accord with a wide range of experimental 
observations of such systems. Indeed, dynamical persis- 
tence is not a novel experience; often hydrodynamic en- 
gineers and plasma experimentalists expend much effort 
in attempts to eliminate persistent chaos. Here we de- 
scribed a mechanism in which the dynamical persistence 
of high-dimensional systems is retained under parame- 
ter perturbation, despite the fact that stricter notions of 
dynamical equivalence are violated. This sets the stage 
for more specific investigations of the statistical topol- 
ogy of stable and unstable manifolds in high-dimensional 
systems — investigations that, one hopes, will lead to pre- 
dictive scaling theories for observed macroscopic proper- 
ties that are grounded in microscopic dynamics. 
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